Extracting Website Data and Parsing HTML Elements

Course: Python Training

Topic: Extracting Website Data and Parsing HTML Elements

Course Link: Python Training Course at JustAcademy

Demo Registration: Register for a Course Demo

Official Website: JustAcademy

1. Introduction to Website Data Extraction

Website data extraction is the process of collecting information from web pages and converting it into a structured format for analysis, reporting, storage, or further processing. Websites contain information such as headings, paragraphs, links, images, product details, prices, reviews, and tables. Python allows developers to retrieve permitted web pages and extract selected information automatically.

HTML (HyperText Markup Language) defines the structure of a web page. A browser interprets HTML to display content, while a Python program can parse the same markup to locate specific elements and extract their text or attributes.

For example, a Python script can extract article titles from a blog, collect links from a documentation page, or read rows from a public table when automated access is permitted. This process is commonly known as web scraping.

Applications of Website Data Extraction

• Collecting article titles, authors, and publication dates.
• Extracting product names and descriptions from authorized catalog pages.
• Gathering links from documentation and educational websites.
• Reading tables containing publicly available information.
• Organizing website content for research and data analysis.
• Monitoring page changes when the website permits automated access.

2. Learning Objectives

• Understand HTML tags, attributes, and nested elements.
• Retrieve web page content using Python Requests.
• Parse HTML using Beautiful Soup.
• Find elements using tags, classes, IDs, and CSS selectors.
• Extract text, hyperlinks, images, and table data.
• Handle missing elements and HTTP errors safely.
• Save extracted information into CSV and JSON files.
• Apply responsible scraping practices and validate extracted data.

3. Understanding HTML Structure

HTML organizes a web page into elements. Most elements have an opening tag, content, and a closing tag. Attributes provide additional information about an element.

<!DOCTYPE html>
<html lang="en">
<head>
    <title>Python Learning Resources</title>
</head>
<body>
    <h1 id="page-title">Python Learning Resources</h1>
    <p class="description">Learn Python step by step.</p>
    <a href="/tutorials/python">View Tutorials</a>
</body>
</html>

Important HTML Elements

Tags, Attributes, and Text

• Tag: Identifies an element, such as p, h1, or a.
• Attribute: Supplies additional information, such as class, id, href, or src.
• Text content: The readable content inside an element.
• Nested elements: Elements placed inside other elements.

4. Installing the Required Python Libraries

Two popular libraries for basic HTML extraction are Requests and Beautiful Soup. Requests retrieves the web page, while Beautiful Soup parses the HTML document.

Install Requests and Beautiful Soup

python -m pip install requests beautifulsoup4

Run the command in a terminal or command prompt using the Python environment for your project.

Import the Libraries

import requests
from bs4 import BeautifulSoup

5. Retrieving HTML Using Requests

The requests.get() function sends an HTTP GET request to a URL. If the server responds with HTML, the response body can be parsed using Beautiful Soup.

import requests

url = "https://example.com"

try:
    response = requests.get(url, timeout=10)
    response.raise_for_status()

    html_content = response.text
    print(html_content[:500])

except requests.exceptions.RequestException as error:
    print("Unable to retrieve webpage:", error)

Explanation

1. url stores the address of the target web page.
2. requests.get() sends an HTTP request.
3. timeout=10 limits how long the program waits for the request.
4. raise_for_status() raises an exception for unsuccessful HTTP status codes.
5. response.text provides the response body as decoded text.
6. The exception handler catches request-related errors.

A successful request does not guarantee that the expected content is present. The response might contain a login page, an error page, or a minimal HTML document. Inspect the response before extracting information.

6. Parsing HTML with Beautiful Soup

Beautiful Soup converts HTML into a navigable structure. The html.parser parser is included with Python and is suitable for many introductory tasks.

from bs4 import BeautifulSoup

html = """
<html>
  <head><title>Python Resources</title></head>
  <body>
    <h1>Learn Python</h1>
    <p>Practice Python every day.</p>
  </body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

print(soup.title.get_text(strip=True))
print(soup.h1.get_text(strip=True))
print(soup.p.get_text(strip=True))

Output:

Python Resources
Learn Python
Practice Python every day.

Common Beautiful Soup Methods

7. Extracting Text from HTML Elements

Use get_text() to retrieve readable text from an element. The strip=True argument removes leading and trailing whitespace.

html = """
<div class="article">
    <h2>Introduction to Python</h2>
    <p>Python is a general-purpose programming language.</p>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

heading = soup.find("h2")
paragraph = soup.find("p")

if heading:
    print("Heading:", heading.get_text(strip=True))

if paragraph:
    print("Description:", paragraph.get_text(" ", strip=True))

Extracting Multiple Elements

html = """
<ul>
    <li>HTML</li>
    <li>CSS</li>
    <li>Python</li>
</ul>
"""

soup = BeautifulSoup(html, "html.parser")

for item in soup.find_all("li"):
    print(item.get_text(strip=True))

Output:

HTML
CSS
Python

8. Finding Elements by Class and ID

HTML classes are commonly used to group related elements. An ID usually identifies one particular element in a document. These attributes help target the correct content.

Searching by Class

html = """
<div class="product-card">
    <h2>Wireless Keyboard</h2>
    <p class="price">₹1,499</p>
</div>
"""

soup = BeautifulSoup(html, "html.parser")

card = soup.find("div", class_="product-card")

if card:
    name = card.find("h2")
    price = card.find(class_="price")

    print(name.get_text(strip=True) if name else "Name unavailable")
    print(price.get_text(strip=True) if price else "Price unavailable")

Searching by ID

html = '<h1 id="main-heading">HTML Parsing</h1>'

soup = BeautifulSoup(html, "html.parser")
heading = soup.find(id="main-heading")

if heading:
    print(heading.get_text(strip=True))

Use class_ when passing a class name to find(), because class is a reserved Python keyword. In CSS selectors, a dot represents a class and a hash represents an ID.

9. Using CSS Selectors

CSS selectors make it possible to locate elements based on tags, classes, IDs, attributes, and relationships between elements.

html = """
<article class="post">
    <h2>Learning Python</h2>
    <p class="summary">A beginner-friendly guide.</p>
</article>
"""

soup = BeautifulSoup(html, "html.parser")

title = soup.select_one("article.post h2")
summary = soup.select_one("article.post .summary")

print(title.get_text(strip=True) if title else "Title not found")
print(summary.get_text(strip=True) if summary else "Summary not found")

10. Extracting Hyperlinks

Hyperlinks are generally defined by the <a> element. Their destinations are stored in the href attribute.

html = """
<a href="https://example.com/about">About Us</a>
<a href="/courses/python">Python Course</a>
"""

soup = BeautifulSoup(html, "html.parser")

for link in soup.find_all("a", href=True):
    text = link.get_text(" ", strip=True)
    href = link.get("href")

    print("Text:", text)
    print("URL:", href)

A link may be relative, such as /courses/python, or absolute, such as https://example.com/courses/python. Python's urljoin() function can resolve relative URLs against the page's base address.

from urllib.parse import urljoin

base_url = "https://example.com/resources/"
relative_url = "../courses/python"

absolute_url = urljoin(base_url, relative_url)
print(absolute_url)

Output:

https://example.com/courses/python

Do not automatically follow every extracted URL. Validate the destination, permitted host, and URL scheme before making additional requests.

11. Extracting Image Information

Image elements commonly use the src attribute to identify an image and the alt attribute to provide alternative text.

html = """
<img src="/images/python-course.png"
     alt="Python training course">
"""

soup = BeautifulSoup(html, "html.parser")
image = soup.find("img")

if image:
    print("Source:", image.get("src"))
    print("Alt text:", image.get("alt"))

Convert a relative image path to an absolute URL when needed:

from urllib.parse import urljoin

page_url = "https://example.com/courses/python"
image_path = "/images/python-course.png"

image_url = urljoin(page_url, image_path)
print(image_url)

Some websites use lazy loading and store image URLs in attributes such as data-src. Inspect the HTML to determine which attribute contains the actual image address. Extracting an image URL does not download the image itself.

12. Extracting Data from HTML Tables

HTML tables use <tr> for rows, <th> for header cells, and <td> for data cells. Beautiful Soup can extract the contents of each row.

html = """
<table>
    <tr><th>Course</th><th>Duration</th></tr>
    <tr><td>Python</td><td>8 Weeks</td></tr>
    <tr><td>SQL</td><td>4 Weeks</td></tr>
</table>
"""

soup = BeautifulSoup(html, "html.parser")
table = soup.find("table")

if table:
    for row in table.find_all("tr"):
        cells = row.find_all(["th", "td"])
        values = [cell.get_text(" ", strip=True) for cell in cells]
        print(values)

Output:

['Course', 'Duration']
['Python', '8 Weeks']
['SQL', '4 Weeks']

For complex tables, consider rowspan and colspan, which affect how cells map to rows and columns. Check the table structure before converting extracted values into records.

13. Handling Missing Elements

HTML layouts may change, or optional content may be missing. Always verify that an element exists before calling methods on it.

heading = soup.find("h1")

if heading is not None:
    title = heading.get_text(strip=True)
else:
    title = "Title not available"

print(title)

Without this check, calling get_text() on a missing element can raise an AttributeError.

Reusable Text Extraction Function

def extract_text(parent, selector, default=""):
    element = parent.select_one(selector)

    if element is None:
        return default

    return element.get_text(" ", strip=True)

title = extract_text(soup, "h1", "Untitled page")
description = extract_text(soup, ".description", "No description")

print(title)
print(description)

14. Complete Example: Extracting Article Information

This example uses sample HTML to extract article titles, summaries, and links. Sample data is useful for practicing parsing without making repeated requests to a live website.

from bs4 import BeautifulSoup
from urllib.parse import urljoin

base_url = "https://example.com/blog/"

html = """
<html>
  <body>
    <article class="post">
      <h2 class="title">Getting Started with Python</h2>
      <p class="summary">Learn the basics of Python.</p>
      <a href="/blog/python-basics">Read Article</a>
    </article>
    <article class="post">
      <h2 class="title">Understanding HTML</h2>
      <p class="summary">Explore HTML tags and elements.</p>
      <a href="/blog/html-guide">Read Article</a>
    </article>
  </body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")
articles = []

for article in soup.select("article.post"):
    title_element = article.select_one(".title")
    summary_element = article.select_one(".summary")
    link_element = article.select_one("a[href]")

    title = (
        title_element.get_text(" ", strip=True)
        if title_element else "Untitled"
    )

    summary = (
        summary_element.get_text(" ", strip=True)
        if summary_element else ""
    )

    href = link_element.get("href") if link_element else None
    url = urljoin(base_url, href) if href else None

    articles.append({
        "title": title,
        "summary": summary,
        "url": url
    })

for article in articles:
    print(article)

How the Example Works

1. Beautiful Soup parses the supplied HTML.
2. The selector article.post locates article containers.
3. Each article is searched for a title, summary, and hyperlink.
4. Missing fields receive fallback values.
5. urljoin() converts relative article paths to absolute URLs.
6. Each article is stored as a dictionary inside a list.

15. Saving Extracted Data to CSV

The built-in csv module can write extracted dictionaries to a CSV file that can be opened in spreadsheet applications.

import csv

articles = [
    {
        "title": "Getting Started with Python",
        "summary": "Learn the basics of Python.",
        "url": "https://example.com/blog/python-basics"
    },
    {
        "title": "Understanding HTML",
        "summary": "Explore HTML tags and elements.",
        "url": "https://example.com/blog/html-guide"
    }
]

if articles:
    with open(
        "articles.csv",
        "w",
        newline="",
        encoding="utf-8"
    ) as file:
        fieldnames = ["title", "summary", "url"]
        writer = csv.DictWriter(file, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(articles)

print("Article data saved successfully.")

16. Saving Extracted Data to JSON

JSON is useful when data needs to be shared with another application or stored in a structured text format.

import json

articles = [
    {
        "title": "Getting Started with Python",
        "summary": "Learn the basics of Python.",
        "url": "https://example.com/blog/python-basics"
    }
]

with open("articles.json", "w", encoding="utf-8") as file:
    json.dump(articles, file, indent=4, ensure_ascii=False)

print("Article data saved as JSON.")

CSV is convenient for tabular data, while JSON supports nested dictionaries and lists. Choose the format that fits the next stage of your application.

17. Extracting Data from a Live Web Page

This example retrieves an accessible web page, checks the response, and extracts its title, headings, and links. Use this approach only on websites you are authorized to access, and follow their applicable rules for automated requests.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://example.com"

try:
    response = requests.get(
        url,
        timeout=10,
        headers={"User-Agent": "LearningHTMLParser/1.0"}
    )
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "")

    if "html" not in content_type.lower():
        raise ValueError("The response does not appear to be HTML.")

    soup = BeautifulSoup(response.text, "html.parser")

    page_title = (
        soup.title.get_text(" ", strip=True)
        if soup.title else "No title found"
    )
    print("Page title:", page_title)

    print("\\nHeadings:")
    for heading in soup.select("h1, h2, h3"):
        print(heading.get_text(" ", strip=True))

    print("\\nLinks:")
    for link in soup.select("a[href]"):
        text = link.get_text(" ", strip=True)
        href = urljoin(url, link["href"])
        print(text, "->", href)

except requests.exceptions.RequestException as error:
    print("Request failed:", error)
except ValueError as error:
    print("Response validation failed:", error)

The user-agent identifies the script, but it does not grant permission to access a website. A production scraper should validate extracted URLs and handle unexpected responses appropriately.

18. Understanding Dynamic Websites

Some websites generate content in the browser using JavaScript. The HTML returned by a normal HTTP request may not include the content visible after the page finishes loading. Beautiful Soup parses the supplied markup but does not execute JavaScript.

If expected information is missing:

1. Inspect the HTML returned by Requests.
2. Check whether the required content is present in the original page source.
3. Look for an official API that provides the information.
4. If permitted and necessary, use a browser automation tool such as Playwright to render the page.
5. Wait for the relevant content to load before inspecting the rendered HTML.

Do not bypass authentication, CAPTCHAs, access controls, or other technical restrictions. Use authorized data sources and documented interfaces whenever possible.

19. Common Errors and Troubleshooting

20. Best Practices for Website Data Extraction

• Check permissions: Review the website's terms, robots.txt guidance, licensing, and applicable laws.
• Prefer official APIs: Use a documented API when it is available and authorized.
• Limit request rates: Avoid excessive requests and unnecessary repeated downloads.
• Use timeouts: Prevent network requests from waiting indefinitely.
• Handle errors: Check HTTP status codes and catch expected exceptions.
• Validate data: Check required fields, formats, and missing values before saving records.
• Choose stable selectors: Prefer meaningful IDs, classes, and semantic elements over fragile positional selectors.
• Organize code: Separate fetching, parsing, validation, and saving into functions.
• Protect privacy: Avoid collecting personal or sensitive information without appropriate authorization and a valid basis.
• Respect content rights: Public accessibility does not automatically mean content is free to republish or reuse.

21. Mini Project: Website Article Extractor

Project Goal: Create a Python program that extracts article titles, summaries, and links from an HTML document and saves them to a CSV file.

Project Requirements

• Retrieve a permitted web page or read a supplied local HTML file.
• Parse the HTML with Beautiful Soup.
• Locate article containers with a CSS selector.
• Extract titles, summaries, and URLs.
• Handle missing elements with fallback values.
• Resolve relative URLs when appropriate.
• Save records in CSV format.
• Display the number of records processed.

Suggested Function Structure

def fetch_html(url):
    """Retrieve HTML from an authorized website."""
    pass

def parse_articles(html, base_url):
    """Extract article data from HTML."""
    pass

def save_articles(articles, filename):
    """Write article records to CSV."""
    pass

Implement each function independently. Test the parser with sample HTML before connecting it to a live website. This makes debugging easier and reduces unnecessary network requests.

Possible Enhancements

• Add command-line arguments for the URL and output filename.
• Support JSON as an alternative output format.
• Log errors to a file.
• Remove duplicate URLs before saving.
• Add tests for empty HTML and missing elements.
• Implement pagination only when permitted by the site's rules.

22. Practice Exercises

1. Create an HTML document and extract its page title.
2. Find all paragraph elements and print their text.
3. Extract headings from h1 through h3.
4. Find elements by class and ID.
5. Extract hyperlinks and their href values.
6. Convert relative URLs to absolute URLs with urljoin().
7. Extract image sources and alternative text.
8. Read an HTML table and convert its rows into lists.
9. Handle a missing element without raising an exception.
10. Save extracted dictionaries to CSV or JSON.
11. Write a reusable function that parses article data from HTML.
12. Explain why Beautiful Soup does not retrieve content generated only after JavaScript execution.

23. Interview Questions and Answers

Q1. What is web scraping?

Web scraping is the automated process of retrieving information from web pages and extracting selected data into a structured format.

Q2. What is HTML parsing?

HTML parsing means interpreting HTML markup and creating a structure that a program can navigate to locate elements, attributes, and text.

Q3. What is Beautiful Soup?

Beautiful Soup is a Python library used to parse HTML or XML documents and search or navigate their element structure.

Q4. What is the difference between Requests and Beautiful Soup?

Requests retrieves web responses over HTTP. Beautiful Soup parses the HTML or XML content that has already been obtained.

Q5. What is the difference between find() and find_all()?

find() returns the first matching element or None. find_all() returns a list of matching elements.

Q6. How do you extract text from an element?

Use element.get_text(strip=True) after checking that the element exists.

Q7. How do you find an element by class?

Use soup.find(class_="description") or soup.select_one(".description").

Q8. How do you extract a hyperlink?

Find the anchor element and read its destination using link.get("href").

Q9. What does urljoin() do?

It combines a base URL and a relative URL to produce an appropriate absolute URL.

Q10. Why might a scraper fail to find visible content?

The content may be generated by JavaScript, the selector may be incorrect, or the server may return different HTML from what the browser displays.

Q11. How should missing elements be handled?

Check whether the search result is None before accessing its text or attributes, and provide a suitable fallback when necessary.

Q12. How can extracted data be saved?

Use Python's csv module for tabular information or the json module for structured JSON data.

Q13. Does Beautiful Soup execute JavaScript?

No. It parses supplied markup. An authorized API or browser-rendering tool may be necessary for content generated dynamically.

Q14. What should be considered before scraping a website?

Review permissions, website terms, automated-access guidance, request limits, privacy obligations, and the rights governing reuse of the content.

24. Quick Revision Checklist

• HTML represents page content using elements, attributes, and nested structures.
• Requests retrieves content over HTTP.
• Beautiful Soup parses supplied HTML.
• find() returns the first match; find_all() returns multiple matches.
• select() and select_one() support CSS selectors.
• get_text() extracts readable text.
• get() retrieves attributes such as href and src.
• urljoin() resolves relative URLs against a base address.
• Missing elements and network errors should be handled explicitly.
• Extracted records can be stored as CSV or JSON.
• Beautiful Soup does not execute JavaScript.
• Responsible extraction requires attention to access rules, privacy, and data quality.

25. Conclusion

Extracting website data and parsing HTML elements is a useful practical application of Python. By combining Requests and Beautiful Soup, developers can retrieve permitted web pages, locate elements with tags and CSS selectors, extract text and attributes, handle missing data, and save results in structured formats.

Begin with sample HTML, test selectors carefully, and organize the program into reusable functions for fetching, parsing, validating, and saving data. As projects become more complex, account for dynamic content, network failures, website access policies, and data quality.

Explore Python Training: View the Python Training Course

Register for a Demo: Register for a Course Demo

Visit JustAcademy: Explore JustAcademy

Additional Learning Resource: Beautiful Soup Official Documentation